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S Objectives 


¢ Explore the tradespace of DAA alerting 
parameters to investigate alerting performance 
metrics of RTCA SC-228 Minimum Operational 
Performance Standards (MOPS). 
— Evaluate the sensitivity of alerting parameters on the 


performance metrics of each DAA alert type: 
preventive, corrective, and warning alerts 


— Evaluate the effect of sensor uncertainty on DAA 
alerting and guidance performance 


¢ Help refine and validate MOPS alerting and 
guidance requirements. 


Ss Simulation Platform 


¢ Airspace Concept 


Evaluation System 
(ACES) and JADEM 
Fast-time Simulation 
Framework 

— Simulate NAS-wide air 


traffic operations of UAS 
and VFR traffic 


Various realistic encounters between UAS and VFR traffic in civil 
airspace 
— Historical cooperative and non-cooperative VFR Traffic 
¢ The 84th Rader Evaluation Squadron (RADES) Data (on April 4, 2012) 
— Proposed UAS Flights 


* 18 different types of UAS missions generated by Intelligent Automation Inc. 


UAS DAA Alerting and Guidance System [JADEM] 
— Higher fidelity surveillance model: Honeywell sensor models 
— Dead-reckoning based DAA alerting logic 
— OmniBands-based guidance logic 


UAS Missions Overview 


* Developed under contract with IAI Inc. 


Mission Types UAS Group Flight Duration Cruise Altitude Cruise Speed (KTAS) Flight Pattern 


Aerial Imaging and Mapping Aerosonde Mk 4.7 40~90 min 300 ft ~3000 ft. AGL 44to51 Radiator-grid pattern 
PNT aOYUrViinva\VKelalixel al ays Shadow-B/Sierra 4k,5k, and 6k ft AGL 74 to 89 Radiator-grid pattern 
Air Taxi Cirrus Cirrus SR22T 20~45 mins 6k-11k 153 to 166 


Flood Inund. Mapping | Aerosonde Mk 4.7 Mk 4.7 


1-4 hrs. : 
Flood Stream Flow Aerosonde Mk 4.7 4,000 ft. AGL 46 to 51 Cre Parente ancien anne 
stream direction 
Three types of pattern: 1) 
Law Enforcement peresonde MEA:/ 3~8 hrs 3,000 ft. AGL 44to51 grid pattern, 2) random, 3) 
/FALCON : 
outward spiral 
W/EValulaatsmer-iuqe)| Global Hawk (RQ4A) 4.5~14 hrs. 5,000 ft. to 35,000 ft. AGL 151 to 321 Radiator-grid pattern 


Point Source Emission Shadow-B 0.5~5 hrs. 3,000 ft. AGL 72 to 80 Round-the-clock 
STo}iIMVACelalineya lars Shadow-B 40~260 mins 3,000 ft. to 13,000 ft. AGL 72 to 93 Radiator-grid pattern 


Tactical Fire Monitoring ScanEagle/Shadow-B 1~1.5 hrs. 3,000 ft. AGL 72 to 75 neni! iene paul followin 
the perimeter of a wildfire 
Traffic Monitoring Shadow-B ~2 hrs. 1,500 ft. AGL 58 to 84 a aki eae 
flight path 


Weather Data Collection Global Hawk 1.5~13 hrs. 5,000 ft. to 35,000 ft. AGL 151 to 321 Radiator-grid pattern 
Wildlife Monitoring Aerosonde Mk 4.7 3,000 ft. AGL 44to51 Radiator-grid pattern 


Ss VFR Intruder Traffic 


¢ VER Traffic Scenario 
— One day in April 2012 


¢ Types of intruders with different equipage 
— Intruders operating under IFR (Not considered in this study) 


¢ tis assumed that ownship unmanned aircraft are equipped with onboard 
radar, active transponder (Mode-S), and ADS-B surveillance system to 
detect the intruder aircraft with different equipage. 


Transponder Equipage Percentage 


Coop-VFR 1 ADS-B Out and Mode-S 82% (OSED: 71%*) 

Coop-VER 2 Mode-S Only 18% (OSED: 14%*) 

Non-Coop VFR No Transponder (or Mode A 15%20% (OSED: 15%*) 
transponder) 


*Based on OSED document: Table A-1. Intruder Equipage Assumptions Post-2020 


Sy Honeywell Sensor Models” 


* Honeywell Sensor Model Version: Version 10 
TCAS Mode-S TCAS Mode-C Radar 


Max = 14 nmi Max = 13.3 nmi 
Detection Max = 90 nmi Max = 30 nmi Vita =) aival cs (FT3) et 


Range (nmi) Min = Onmi Min =O nmi Raines | 
by quadrant) Min = 1000 ft 


Baer +/-180deg | +/-180deg +/- 180 deg +/-135 deg 
' +20 deg +20 deg 
Elevation (deg) +/-20 deg Ales Modes +/-20 deg 


Relative +/- 3000ft +/- 3000ft 


N/A 


Altitude (ft) (Target of Interest) (Target of Interest) 


¢ The parameter values for sensor models were set based on Flight Test 3 (FT3) data. 


Honeywell Tracker 


Tracker Development/Versions 


a NASA ACES Simulations 
and updates 


e UAS in the 
NAS FT4 


Performance review 
and updates 
e 2014 ACAS- 


Xu Flight 
Test 


¢ 2013 Self 
Separation 
Flight Test 


Performance review 
and updates 


e UAS in the 
NAS FT3 
Test 


Performance review Performance review 
and updates and updates 
LCL NASA ACES Spot-Check 


Honeywell has been developing DAA trackers for several years 
— Development and flight testing since 2013 


Results presented here are based on a version used in FT3 
— Results spot checked with FT4 version 


Performance review 
and updates 
e ACAS-Xu 


Run 3 Flight 
Test 


e ACAS-Xu 
Run 2 Flight 


Experimental Plans for 
DAA Alerting Performance Study 


aterting Thresholds (SC-228 MOPS) 


sacra Corrective Warning 
Alert Type | | Se 


a 


Minimum Average 55 55 5 
Time of Alert (Seconds) 7 
Hazard Zone Late Threshold 0 0 15 

Alert Times (THR ate) (Seconds) = 
Early Threshold 
75 75 55 


Non-Hazard 
Zone DMOD and HMD*(NM) 1.5 15 | 


S Zones Used in Alert Evaluation © 


Hazard Zone (HAZ) 


May Alert Zone 


Non-Hazard (HAZNot) Zone 


S Unmitigated Simulation Experiment 


¢ Independent Factors 
— Parameters for alerting logic 
1) Alert time before HAZ (sec): Predicted time to Loss of Well Clear 
(LOWC 
2) HMD/DMOD threshold (nm) 
3) Vertical separation threshold (ft) 


— Existence of sensor noise 
¢ Noise-free tracks vs. tracks from Honeywell sensor model 


¢ Dependent Variables 
— MOP%9’ alerting performance metrics (only for unmitigated 


simulation results) 
¢ Required/early correct alert, short alert, late alert, missed alert, 


correct/early permissible alert, permissible non-alert, incorrect alert, and 


correct non-alert 
¢ Analysis of alert times: Average time of alert 


@c-228 Alerting Performance Metrics 


Performance metrics analyzing tracks with HAZ Violation (Alert 
Required) 

— Probability of Missed Alert: P(MA) 

— Probability of Late Alert: P(LA) 

— Probability of Short Alert: P(SA) 

— Probability of Early Required Alert: P(ERA) 

— Probability of Correct Required Alert: P(CRA) 

— Average Time of Alert Before HAZ violation (AAT) 

— Alerting Ratio (AR) 


Performance metrics analyzing tracks with MAZ violation (Alerts 
Permissible) 

— Probability of Permissible Non-Alert: P(PNA) 

— Probability of Early Permissible Alert: P(EPA) 

— Probability of Permissible Alert: P(PermA) 

— Average Number of Alerts Issued per Encounter (ANA) 


Performance Metrics analyzing tracks in Non-Hazard Zone (Alerts 
Undesirable) 

— Probability of Correct Non-Alert: P(CAN) 

— Probability of Incorrect Alert: P(IA) 


Independent Factor 


Alerting Time 
Threshold (sec): 
Predicted Time to 
Loss of Well-Clear 
(LoWC) 


HMD/DMOD (nm) Same For All Alert Types 
0.66, 0.90, 1.15, 1.5 nm 


Vertical Separation 
Threshold (ft) 


Sensor Uncertainty 0: No Sensor Uncertainty 
1: Sensor Uncertainty 


* Total 72 runs of unmitigated simulations = 3X 4X3 X2 


Final Experiment Matrix 


: Alerting Time Threshold (sec) HMD/DMOD Threshold (nmi) Vertical Separation Threshold (ft) 
rin Honeywell Sensor 


Preventive Preventive Noise 
Alert aes 


0.90 
65 0.90 
35 1.15 
50 1,15 
65 1.15 


20 35 
35 150 
20 35 


50 1.50 
= 1.50 


6! 
6 
6 
6 


35 


Ss 
Ss 
S 
5 
50 
5 
S 
S 
S 


20 
35 


50 65 65 115 600 600 
50 65 65 1.15 750 750 


35 50 
35 50 


65 
35 
50 
65 


3. 35 115 600 600 
5 5 50 1.15 600 600 
65 1.15 600 600 
E! 35 1.50 600 600 
S 5 50 1.50 600 600 
E: 
5 5 
3 
5 


50 1,15 
65 1,15 


20 5. 
3! 0 
50 5 
5 


eat 20 35 35 1,50 750 750 1000 
35 50 50 1,50 750 750 1000 1. 3 
50 65 65 1,50 750 750 1000 
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Data Analysis of Unmitigated Simulations 
with Honeywell Sensor/Tracker Models: 
DAA Alerting Performance Metrics 


S Encounter Definition 


¢ An encounter initialized on 25 nm horizontal x 
6000 ft vertical cylinder, 


AND 


¢ The intruders are predicted to be within 10 nm x 
2000 ft cylinder in the next 5 minutes look-ahead 
time window based on dead-reckoning trajectory 
prediction. 


Percentage of Encounters (%) 


100% 


90% 


80% 


70% 


60% 


50% 


40% 


30% 


20% 


10% 


0% 


Analysis of Encounter Set 
(Proportion of Worst Zone By Each Alert Type) 


HAZ 


HAZNOT 
Worst Zone 


MAZ 


Alert Type 

Mi Preventive Alert 
Mi Corrective Alert 
Mi Warning Alert 
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Analysis for DAA Corrective Alerts 


Independent Factor 


Alerting Time / Preventive Alert 
Threshold (sec): 
Predicted Time to 
Loss of Well-Clear 


tow = 


HMD/DMOD (nm) Same For All Alert Types 
0.66, 0.90, 1.15, 1.5 tm 


Vertical Separation 
Threshold (ft) 


Sensor Uncertainty Py aeees Te No Sensor Tree 
1: Sensor Uncertainty 


* Total 72 runs of unmitigated simulations = 3X 4X3 X2 
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aa Analysis for DAA Corrective Alerts (Ge 


Performance Metrics Analyzing Tracks 
with HAZ Violation (Alerts Required) 


¢ Probability of Correct Required Alert (CRA) 
¢ Probability of Missed Alert (MA) 

¢ Probability of Late Alert (LA) 

¢ Probability of Short Alert (SA) 


Corrective Missed Alert 
Effect of HMD/DMOD Threshold) 


, HMD/DMOD Threshold 
0.013 ° 0.66 
° 09 
e 1.15 
° 15 
e@ 
0.010 
i?) 
aS) 
oO 
= e 
oO 
2 0.008 
Oo 
E 
[e) 
5 
ra e e 
Ke) 
> 0.005 
oi 
[av] 
ol ) 
o Ps 
0.003 . 
e@ 
@ e 
e e 
0.000 
450 600 750 450 600 750 450 600 750 450 600 750 
0.66 0.9 1.15 1.5 


50 
Vertical Separation Threshold (ft) / HMD (nm) / Alert Time Threshold (sec) 


To reduce the P(MA), it would be better to have a larger HMD/DMOD 


threshold and a larger vertical separation threshold. 
20 


Corrective Missed Alert 
Effect of Sensor Noise) 


Sensor Noise 
|e No Sensor Noise 
e Sensor Noise 


0.013 


0.010 


0.008 


0.005 


Probability of Performance Metrics 


0.003 


eee : 450 450 450 450 450 450 450 450 


0.66 0.9 ‘ 1.15 1.5 0.66 0.9 1.15 1.5 
50 50 
Vertical Separation Threshold (ft) / HMD (nm) / Alert Time Threshold (sec) 


The probability of MA is higher across all HAD/DMOD threshold settings 


when there is sensor noise, compared to when there is no sensor noise. 
21 


Corrective Missed Alert 
Effect of Alert Time Threshold) 


Alert Time Threshold 
0.013 ° 35 
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Vertical Separation Threshold (ft) / HMD (nm) / Alert Time Threshold (sec) 


There is no significant effect of DAA alert time threshold on the P(MA). 
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Sy Probability of Missed Alert (MA) 


HMD and Vertical Separation 
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Vertical Separation Threshold (ft) / HMD (nm) / Alert Time Threshold (sec) 
There is a significant effect of HMD and vertical threshold settings and sensor 
noise on the P(MA). On the other hand, there is no significant effect of alert 
time threshold on the P(MA). 23 


Corrective Late Alert 
Effect of HMD/DMOD) 


HMD/DMOD 
0.45-———— , © 0.66 
* 09 
© 1.415 
0.40 2 15 


0.35 


0.30 


0.25 


0.20 


0.15 


Probability of Performance Metrics 
e 


0.05 e 


9:00 450 600 750 450 600 750 450 600 750 450 600 750 


0.66 0.9 1.15 1.5 
- 50 
Vertical Separation Threshold (ft) / HMD (nm) / Alert Time Threshold (sec) 


To reduce the P(LA), it would be better to have a larger HMD/DMOD threshold 
and a larger vertical separation threshold. 
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Corrective Late Alert 
Effect of Alert Time Threshold) 


HMD/DMOD 
e 35 
e 50 
0.24 | e@ 65 
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Vertical Separation Threshold (ft) / HMD (nm) / Alert Time Threshold (sec) 


There is a significant effect of DAA alert time threshold on the P(LA). 
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Corrective Late Alert 
Effect of Sensor Noise) 


Sensor Noise 


¢ No Sensor Noise 
e Sensor Noise 
0.24 


Probability of Performance Metrics 


0.12 

ee 450 600 750 450 
0.9 
35 


There is a higher probability of Late Alerts when there is sensor noise, 


compared to there is no sensor noise 
26 


Probability of Performance Metrics (LA) 


HMD and Sensor Noise 

° 0.66 

°- 0.9 

Ses 

Ca is) 

e Without Sensor Noise 
@ With Sensor Noise 


Vertical Separation Threshold (ft) / HMD (nm) / Alert Time Threshold (sec) 
There is a significant effect of HMD/DMOD, vertical separation, and alert time 
thresholds on the P(LA). On the other hand, there is a less effect of sensor 
noise on the P(LA). 7 
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There is a significant effect of alert time threshold, HMD/DMOD, and sensor 
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noise on the P(CRA). 
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There is a significant effect of alert time threshold, HMD/DMOD, and sensor 
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S Probability of Early Required Alert (ERA) 


HMD and Sensor Noise 
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Vertical Separation Threshold (ft) / HMD (nm) / Alert Time Threshold (sec) 
There is a significant effect of HMD/DMOD, alert time threshold, and vertical 
separation threshold on the P(ERA). 
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Missed Alert 


Alert Performance Metrics 
Ml Correct Required Alert 
Ml Early Required Alert 
Late Alert 
Short Alert 
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Performance Metrics Analyzing Tracks 
with MAZ Violation (Alerts Permissible) 


¢ Probability of Permissible Alert (PermA) 
¢ Probability of Early Permissible Alert (EPA) 
¢ Probability of Permissible Non-Alert (PNA) 
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There is a significant effect of HWD/DMOD, alert time threshold, and vertical 
separation threshold on the P(PermA). 
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Performance Metrics Analyzing Tracks remaining 
in Non-Hazard Zone (Alerts Undesirable) 


¢ Probability of Incorrect Alert (IA) 
¢ Probability of Correct Non-Alert (CNA) 
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There is a significant effect of HMD/DMOD and alert time thresholds on the 
P(IA). 
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There is a significant effect of sensor noise on the P(IA). 
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There is a significant effect of vertical separation and alert time thresholds on 
the P(IA) when there is sensor noise. 
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There is a significant effect of Sensor noise, HMD/DMOD, vertical separation, 
and alert time thresholds on the P(IA). 
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There is a significant effect of HWD/DMOD and Alert Time thresholds on the average time of 
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Analysis for DAA Warning Alerts 


Independent Factor 
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* Total 72 runs of unmitigated simulations = 3X 4X3 X2 


© Analysis for DAA Warning Alerts 


Performance Metrics Analyzing Tracks 
with HAZ Violation (Alerts Required) 


¢ Probability of Correct Required Alert (CRA) 
¢ Probability of Missed Alert (MA) 

¢ Probability of Late Alert (LA) 

¢ Probability of Short Alert (SA) 
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There is a significant effect of HMD and vertical threshold settings and sensor 
noise on the P(MA). On the other hand, there is no significant effect of alert 
time threshold on the P(MA). 49 


Probability of Performance Metrics (LA) 
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There is a significant effect of HMD/DMOD, vertical separation, and alert time 
thresholds on the P(LA). On the other hand, there is a less effect of sensor 
noise on the P(LA). 50 
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There is a significant effect of alert time threshold, HMD/DMOD, and sensor 
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There is a significant effect of HMD/DMOD, alert time threshold, and vertical 
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All Performance Metrics 


Alert Performance Metrics 
e P(CRA) 
e P(ERA) 
e P(LA) 
e P(MA) 
e P(SA) 
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© Analysis for DAA Warning Alerts 


Performance Metrics Analyzing Tracks 
with MAZ Violation (Alerts Permissible) 


¢ Probability of Permissible Alert (PermA) 
¢ Probability of Early Permissible Alert (EPA) 
¢ Probability of Permissible Non-Alert (PNA) 
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© Analysis for DAA Warning Alerts 


Performance Metrics Analyzing Tracks remaining 
in Non-Hazard Zone (Alerts Undesirable) 


¢ Probability of Incorrect Alert (IA) 
¢ Probability of Correct Non-Alert (CNA) 
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Average Time of Alert before HAZ Violation (sec) 
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There is a significant effect of HWD/DMOD and Alert Time thresholds on the average time of 
alert. Vertical separation threshold has an effect on the average time of alert as well. 
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Ss Summary of Analysis (1) 


¢ Overall, HMD/DMOD threshold has significant effect on the performance 
metrics. 


¢ There is a noticeable effect of HMD/DMOD, vertical separation threshold, 
and sensor noise on the probability of Missed Alert, P(MA). On the other 
hand, DAA alert time threshold has less effect on the P(MA). 
— To reduce the P(MA), it would be better to have a larger HMD/DMOD threshold and a 
larger vertical separation threshold. 
¢ There is a noticeable effect of HMD/DMOD, vertical separation, and alert 
time thresholds on the probability of Late Alert, P(LA). The HMD/DMOD 
threshold has the most significant effect on the P(LA). On the other hand, 
there is a less effect of sensor noise on the P(LA). 
— It would be also better to have a larger HMD/DMOD threshold, a longer alert time 
threshold, and a larger vertical separation threshold to reduce the P(LA). 
¢ There is a noticeable effect of HMD/DMOD, vertical separation, and alert 
time thresholds on the probability of Early Required Alert, P(ERA). Alert 
Time threshold has more effect on the P(ERA) than on the P(LA). With a 
longer Alert Time threshold, it will increase the P(ERA) significantly while 
reducing the P(LA) a little bit. 


¢ Alert time threshold and sensor noise have a noticeable negative effect on 
the probability of Correct Required Alert, P(CRA). On the other hand, 
Vertical separation threshold has less effect on the P(CRA). 
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Ss Summary of Analysis (Il) 


The probability of Permissible Alert, P(PermA), is also significantly 
affected by the HMD/DMOD threshold, alert time threshold, and 
vertical separation threshold. 


There is a noticeable effect of sensor noise, HMD/DMOD and Alert 
Time threshold on the probability of Incorrect Alert, P(IA). 
HMD/DMOD threshold is much more sensitive factor to the P(IA) 
than the Alert Time threshold. 


There is noticeable effect of HMD/DMOD and Alert Time thresholds 
on the average time of alert. Vertical separation threshold has an 
effect on the average time of alert as well. 

— To meet the MOPS requirement on the minimum average time of alert for 
DAA Corrective Alerts (55 seconds), the HMD/DMOD threshold should be at 
least 0.9 nmi and alert time threshold would be better to set at least 50 
seconds before HAZ when there is sensor noise. 

— Our results show that the requirement on the minimum average time of DAA 
Warning Alerts (25 seconds) can be satisfied even without any buffer on 
HMD/DMOD and with a shorter alert time threshold (35 seconds). 

— However, if HMD/DMOD threshold is larger than 1.15 and Alert Time 
threshold is larger than 50 seconds, it will increase the probability of Early 
Required Alert, P(ERA), significantly for DAA Warning alerts. 


Ss Future Tradeoff Study 


- Investigate the relationship between late/missed alerts 
and incorrect/early required alerts 
— Can we reduce the number of incorrect or early alerts without the 
immediate expense of higher late/missed alerts? 
— We might be able to reduce missed and late alerts with larger 
buffers, but it will increase the number of early or incorrect alerts. 
— Can we increase average alert time (AAT) without the expense 
of too many Early or Incorrect alerts? 
¢ Further research on the effect of DAA Permissible alerts 
— Effect of the Permissible alerts on UAS pilot’s performance 
— Permissible alerts might have more negative alerts than the early 
required alerts since the early required alerts are still correct 
alerts. 


— It might be desirable to reduce the number of all permissible 
alerts including early permissible alerts as well as the number of 
incorrect alerts. 
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Sy Backup Slides 
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Parameters Values DYekYol gl eldtel aly 


Modified Tau Ratio of range to range rate with DMOD 


DMOD 4,000 ft Distance modification that represents a minimum 
desirable range between two aircraft 
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S VER Traffic (courtesy of 84‘" RADES) 


The 84th Radar Evaluation Squadron (RADES) data were used. 


— There is no explicit information that could be used to determine 
whether radar hits come from IFR flights or VFR flights. 


— Asingle aircraft could be detected from multiple radar sites. 


¢ Cooperative VFR tracks were processed using 


— Aclustering method based on a modified minimum spanning tree 
algorithm, 


— Filters to categorize each track into IFR or VFR: altitude, speed, and 
Mode 3 code. 


¢ Non-cooperative VFR tracks were processed 


— Using algorithm developed by Honeywell to process non-cooperative 
VFR tracks and estimates altitude measurements 


VFR tracks were smoothed using the IAI’s smoothing method 
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Sample of Processed VFR Traffic: 
Geographic Flight Density 


0.03 0.07 O14 O29 062 130 2.73 5.75 12.11 25.51 


Average counts of VFR flights flying simultaneously 
within 1x1 degree blocks (latitude/longitude) 


* Cooperative VFR Flight Profiles from April 4%, 2012 7 


